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Following a brief history of the Title i a haneion 
Anda Reporting System (TIERS) and the development of the RNC models 
and technical assistance centers, plans are described for activities 
to increase the usefulness of the evaluation data fcr local decision 
‘paking. Three major kinds of activities are explained: projects 
developed by an Evaluation Use Committee; contracts tc. support state’ 
“refinements of ‘TIERS; and the continuing development of modeis 
appropriate for evaluating other Title I areas, particularly ~ 
pre-second grade programs. The Evaluation Use Committee is engaged in 
producing materials on formative and summative evaluation which will 
be disseminated through literature reviews, werkshors for state and \ 
Jocal education agency evaluators and administrdtors, case studies, 
-suggestions for evaluators, and an annotated bibliography. Contracts. 
“to support state refinement of TIERS are divided into four ~ 
categories: improvement of data collection and analysis activities; 
developmemt and' dissemination of new saterials; projects related to ‘ 
testing and evaluation methods; and a. niscellaneous gategory. 
‘Materials and strategies suitakle for improving and "evaluating. 
“ pre-second ade Title’ I ‘programs are also being developed for : 
dissenina ti. (MB) : 
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Y She Waar Title T Balaton Wea an 
Local Decision-Making -~A Federal Perspective 


Judith I. Anderson, USOE/OED - 
Ronald L. Fishbein, USOE/0ED 
Robert “f Stonehill, ‘USOE/OED 


"The Committee repeats its view that the Office ‘of Education has - 
- not assumed the leadership it should have concerning evaluation >— 
~ activities under Title I° ,.. The Committee urges OE to correct 
this ead Tinge® (Stbcommittee on Education, 1974). 


Such. were the feelings of the Committee on Education and Labor 
of. the House of € Representatives in hearings that preceded passage 
of the Education Amendments of 1974." Those feelings resulted in 
van amendment (section Val, of Pals aacaag) ah Title I of the Elementary 
) and Secondary Education Act (ESEA) of 1965; an amendment which has greatly: . 
"inf wenced the course ‘of evaluation in Tit e I programs. In 1977, the 
General Aecqunt ing Office (GAO) publishéd a report titled. “Problems and” 
Needed Improvements in Fielunting office of Education Programs." GAO 
urged. that, - ae ‘ 2 
. "USOE should more strongly pore serving congressional needs. | 
‘in planning and carrying out evaluation studies, should define its 
program objectives more clearly, and should improve the Implementation 
of evaluation resutts." (GAQ,1977) | ‘ 
“The goal was clear, even if the’ strategies which should’ be adopted 
to achieve this goal were less sq -- improve the validity and utility of 
Title I eva uat iog“infornat ion, at the.Federal, State and Tocal level. 
" Background ace C07 oe _ 3 
Title T of ESEA. Seite that the policy of the United States will be 


- 


‘ “to provide ftmancial absistance ... to local educational agencies 
serving areas with concentrations of children from low-income 
families to expand and improv@ their educational programs by 

: various means ... which contribute particularly to meeting the 
special educational needs of educationally deprived children.": 
(ESEA, Title I, section 101, “Declaration of Policy") 
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, In the years since its enactment, the annual appropriation has 
grown from $959 million to approximately $3 billion, and funds have 


been used in ,increasing proportions-to provide remedial services 


“i 


ih the bdsic skills (vize, reading, ldnguage ‘arts and mathenatics) to 
low achieving dtudants (Wargo, Tallmadge, Lipe, and Morris, 1972; 
Trismen, Waller, and Wilder, 1976; and NIE, 1977). | 
Evaluation of the impact of Title services is Peauithad at all) 
three programatic levels - local, State, and Federal. As stated in . 
section 124(4) ar Title I, a local educational agency (LEA) may receive i i 
funds only if : 
“ef fectivé procedures are sdcated Fi evaluating ... the eFiect tans 
ness of the programs assisted under this title in meeting the 
special educa tiona | needs of educationally deprived. children." 
Each State educational agericy (SEA) is required to report periodically ° 
". to the U.S. Conmissiongr of Education (C8nmissioner) the results of ‘the: | 
LEA evaluations, conducted ‘in accordance with an approved schedule: In 
dy the Commi ss oner ‘is Renu to report to Congress eteunisily con- 
we the results of evaluat fons of the Title I program. as ? be ap 
Pasteattempts at the Federal level to use data from the State eval-, 
_ Yatton reports to report to Congress have been frustrated by ied Jack . 
of comparability and Somet tines lack, of validity of ane data ‘i these 
reports. Both the United States Office of Education ‘iso and the .- 
. House Education and Labor Committee recognized these problems, 
and in August, 1974, Congress amended-Title I with the addition © 
of section 151. In November, 1978, Congress again amended ESEA,- 
with section 183 of P.L. 95-561 replacing section 151.- While the 
basic requirements for evaluation were retained, some modifications 
were made to the law. The evaluation requirements can be sum@® 
o - SP .% + G: 
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marized in the following provisions of ithe law: 


; 0 independent evaluations which describe and measure the impact of 
Ro Progra ns and projects, pe : ; rte 
0. development and publication, of deafened for evaluation 
’ of program or project effectiveness, and the .devel opment of 
an evaluation schedule within each SEA so that. representative 
samples of LEA Title I programs are evaluated, each year, a 


0 provision of joint USOE/SEA evaluation studies, 


4 o development of models for evaluation, including uniform procedures 


and- criteria to be utilized by LEAs and SEAs, which produce data 
- which is comparable on a statewide. and’ nationwide basis, 


f-. 3 ‘echnten’ and other assistance to eranle SEAs and LEAs to apply 
{ the evaluation models. ; 


( =~ Implementation of the Legislative Requirements ‘ 
3 . Section. 185 has been’ interpreted to. place priority .on assisting — 
( ; "SEA ‘i LEAs to implement . methodologically sound evaluation models 
| which wil] produce data which are “comparable, and thus aggregatable, 
on a statewide and nationwide basis. The work to develop standards 
_ and models for local and State evaluation of fitle I was begun 
in June, 1974, a few months before the actual legislative mandate for 
this. work was passed. A contract was awarded, through Federal competitive 
procedures, to RMC decinesh Corporation of Mountain View, California. | 
‘That work involved: ° ) . ee” cas s a 


ft) intenviews sith’ pol icy-nakers in'both the plein nd Legista 
tive/Branches to determine their information needs, 


o a review of all State. Title I evaluation reports for the previous 
five year period, ‘ 


“9 a tabulation of available evaluation data ivoire the Title I program; 


0 a determination of the Common’ reporting practices “nich might ‘be , 
adopted , hati ngioe4 to sarvice: policy-makers' needs, 


o the recommendation pe evaluation and reporting practices, 


0 “a check on the feasibility of ‘the suggestions wth administrators 
and evaluators ina small. ‘samp le- of states el Sau 
Wood, and Binkley, 1975). , . fe eee 


/ «In dune ,. 1975, this: devel opmental effort was cbentetd, and another 
}e 


award was made to RMC Reseach Corporation ERrOUgT. competitive. ‘ies 


cedures. That project (Involved visits to all States and Garvitorias, 


as Well as to three LEAS in each, to discuss the pratotype system that — 
had been developed and its Pepiicattons in Vocal Settings (Bessey, pat 
Chiang, and Tallmadge, ier ‘ ; | ace ae 


* 


Throughout both projécts, te advisory groups were relied upon for 


eiiamiie: “and feedback. the. Policy Advisory Group consisted. 7) 
of SEA Title I prograni adninistrators and evalyators, LEA program’ 


administrators and evdluators, parents, ahd representatives of both 


Ls the National Advisory Council on the Education of Disadvantaged a 7 
— Children and the Council] of Chief School Officers. The Research Advisory ih 
Group included nationally recognized experts in the reas of evaluation / 
: and:measurement’. Both panels reacted to plans, Savaaetel changes, ~ . } 


and provided Advice through all phases ‘of the work. ‘Comments were also if 
Vv , 9 f 
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solicited from SEA and LEA administrators. during the site visits 
~ and have been sou ght subsequent ly on an informal basis from States and 
local administrators invélved in trying’ out the newly devel oped evaluation 
procedures. ° | - ie & an i 
. al 
The resulting Title I Evaluation and.Reporting | System sae aac 
= 4s associated reading, nathena ies and language arts evaluation models, : 


“suitable for use in grades 2 - A2, are wel 1l-documented and will ngt 


‘ . > 


be discussed in detail here. However’, it should be noted, that } &. 
the procedures are essentially refinements of comonly used-practices. 
a} 
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* of ‘deriving, the expected iseebh est performance (no-treatment expectation). 


\ ‘ 


_. to the State and the kinds of .services that will be provided: -Within 


A! 


Each of the three aodlele specifies procedures far conan. the tast’ re 


2 Rormnnge of the Title I group at the bnd of the POGeAMs with an éstima e 
of what that. performance would have been without Title I for. Title’ Ls) 
in combination-with other conpensitory educational services) apectal, 


eunplenentiry sayviies: The windite basically differ in their: method. 


To meet the priority of assisting States: in jnolanenting’ the: Title I 


evaluation modé!s, USOE personnel haye developed a technical assistance 


“ 
program. IA October, TY76 the Title I Syalietion Technical Assistance 


Centers (TACs), the largest. component of the assistance program, both’ 
in terms of dollar expenditure: and person-hours of effort, were 
funded. - Other components of the, USOE technical assistance program } 


include the periodic Brow haven of regional workshops on newly developed 


portions of the TIERS | (such as the 9 workshops conducted in the 


Fall of 1976 by RMC to share the nisrlyisdahetsiad reguiar Title I . 4 


‘ program evaluation’ models), and a publications: series including 


monographs, technical panphtets and newsletters. 

The ten (one for each. HEW regional area) TACs seavita to SEAs 
and LEAs free consulting services to aid in ‘the evaluation of their .° 
Title I programs. Each TAC has devel oped an agreement with its client® 


’ SEAS concerning “the conditions sind which it will provide services , " o 


the constraints required.to be sure that a State is proceeding in 


a positive manner to meet the requirements of the law, each State 


¢ agency haqbeen able to determine the areas of consulting services 


to be emphasized by the TAC within the State. ; ” 
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Meeting Legislative. Priorities 
- The requirement that LEAs, use. evalvation <aiestarts whtcl produce 
data neeavalts on a statewide and nationwide basis places dontes ints 
* upon the LEAS" ‘choice of procedures. RNC Research Corporation, if 1ts.° 
devel opment of the acdets.etheantan to meet the data comparabilAty 
requirement of the legislation and, at the same time, provide a 
to LEA and SEA decision-makers an evaluation system which would’ 
yield useful data. This was a worthy goal, and. one endorsed by te 
; Congress, but unfortunately it proved virtually impossible to meet the 
information needs of each of the three Tevels of decision-makers (1oca1, 
State and Federal) with one set of procedures. When this became 
“apparent, USOE Annes made the ‘choice to ‘develop evaluation models 
and accompanying report forms vee would provide Congress with the iter 
formation ‘that it’ requested, and to provide eddbeiona) assttance to SEAS 
and LEAs to meet any additional needs through a pee assistance 
program. = . oa 
-Sirice September, 1976, USOE personnel have placed first stake 
on assisting States to implement the reading, nathenat ics and 
language arts aya loation models. Without a secure foundation on which . 


to base. more elaborate ‘and ambitious evaluation strategtes, it would have 


proven extremely eT enIE: to organize and disseminate salaries acti-. 


vities aimed at ‘local program ‘improvenent . However, it should be noted—— 


that although USOE has placed priority on the implenentat ion”of the TIERS, 
‘assistance’ In areas of TWele I evaluation which are not. addressed” by these 
models has ean provided to some SEAs and LEAs, and the svadlabitity of 
ach services. Has beén made generally known. 
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4a : ‘The model's for eeuetires the achievement impact. of ritle I progtags, 

| have been publ ished in the: Federal Register {February 7 1979), as draft 
P regulations. The hope is that through application of these models. at the : 


J. SEA and LEA. levels, USOE wil] be provided with data that can‘be used to- 


develop a ma jor-component of 1 report to ‘Congress on the status 


( 


of the Title I program. / ; 
4 Now that all SEAS are preparing to implement the proposed 

evaluation riodels , and many have already done sa on a. statewide basis. : \ 

for heli ail ae can devote more effort to helping States make | 

better use of the data that: is being produced through the implementa fon 

of the TIERS and to assist any SEAs or LEAs to develop ava Tuatian 

*, Strategies that will address quest fons of Program importance and interest 

that are not adequately answered by implementing the TIERS. In ‘the 

next aan of this paper, USOE efforts in this area will, be - | 


" discussed. mye ee 


. 
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Activities to Increase the Utility of Evaluation Data , 


Activities to support local evaluation use are Underway jn three 
major areas, with the services provided by the TACS, and: the funding i 
of fourteen SEA- initiated’ "State Refinements to the “USOE. Title I Evaluation 
System" the most prominent. 
An Evaluation Use Comittee charged with the task of investigating 
Z- methods of tnerees ing the local utility of Title I evaluation data, has 
been in soeration since the Fall of 1978. The Committee, which is 
composed of TAC staff who are particularly interested’ in the atility 
problem, was OR gEOLERG by USOE in-response to requests from -SEAs and . 


LEAs who wanted to, enhance the value both of data collected as ere 
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of the TIERS aicten as well as that collected, as Kart. of a mre general 


rogram ‘evaluation framework. The current effort is aimed primarily at 


| or SEAs and EAs ‘who have reqested assistance in this area, although . 


Ma 
- 
. 


attempts are being made to incraase oeee arnong those program admini- — 


stratoré who have -not routinely been ustng their evaluation results for 


program ‘review, planning and improvement . often, ‘interest sin data use has 


i ‘evolved from a ‘previous invol¥enen€in other: areas necessitating the 
development. of a Sound uiiatian framework , such ‘as the srapara elon for 
-a submission for exemplary praJect certification ‘through the Joint 

po igeenine sion Review Panel (adap) 

. » The Eva luat {on Use Committee thas-taken‘a eo-facted appranch 
determined tn part by’ phil asophica] differences among ‘Committee members. 
“One subgroup is concentrating their efforts on product evaluation seh 
“primarily TIERS data, which includes some paekgyoune atoject descriptors 
and estimated NCE ais -- while the other subgroup is. focusing on 
process evaluation. The two subgroups will combine their work into an 

‘ integrated report, i neludi ng. instructional materials, which will 

be produced under the coordination of the Region V TAC Directors. These 
materials will represent a synthesis of work done by all of the TACs. 
The goals of this project are to develop an orientation towards asking 


meaningful questions about programs, to emphasize the need for quality 
4, 


data, to show how evaluation data generated from the TIERS can appropriate- 


ly be interpreted, to illustrate, the planning and implementation of’a 
‘formative evaluation strategy, and to provide actual examples 
of how evaluation data have ‘been used in decision-making. 

The materials Preeuene and cor lated by the Committee will consist 


of a literature review of articles, books, or reports on evaluation use, 


p 
i0. ; 
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_ workshops for LEA and SEA evaluators -and administrators; case, studies 


suggestions ‘for evaluators, and an annotated bibliography. The, a. 


| 


i 
: ‘literature heviee will cover methods ‘of vee as well as providing 


ane the reader with examples of. producti ve evaluations. . | 
An evaluation use workshop series, aimed. at. " encoyraging LEA use > 
of TIERS data. is under devetounent « The primary method used ‘is the 
- presentation of vignettes, each of which raises questions about the 
meaning of the evaluation results presented. . Four major chtegories _ 
; of evaluation data use are identified: TApEReING evaluation procedures, . 
Title I program decisions: informing interested Spode and within-project 
. planning decisions. | While the workshop “series is intended to enhance 
the production and employment of valid and accurate TIERS data at the 
_ district level, the need for additional information beyond what is 
routinely cot lected: through the TIERS is also emphasized. In addition, al 
* the “werishoe highlights include data quality issues that are known “to 
affect evaluation results, such as the accuraty of score conversions 
‘and analysis. Staff at the Region II TAC are also developing a workshop 
. package, with the emphasis. to be on the SOR TeMN TA ELON of process. 
‘evaluation as an adjunct to the TIERS. 
One workshop to be tncJjuded will cover the area of effectively 
using Title I evaluation data for local decision-naking. One segment 
. considers various definitions and perceptions of evaluation, while ae, ae 
ae another shows the relationships between program Saaniind and: avaliation 
| planning, program implementation and evaluation implementation, and: the 
ultimate role of evaluation in program modification. A separate segment 
considers appropriate data and presentation frethods for reporting and. * ne 


*. effectively communicating test or evaluation results to various audiences. 
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The materials package will also contain case studies. of actual 
‘Title 1 ‘project evaluations’ +acoRder to. provide information on pabiens 
‘watch may be encounneredy as well as. to provide examples of ‘successful 
evaluation stratedies. ‘Up ‘to. now,. most LEAS (though certainly not all) : 
.” have Focused their evaluation efforts on the Federal ly-required activities 
outlined in the proposed ‘page viens: However, interest goes far beyond — = 
this, particularly as the implementation of the required evaluation . 
activities becomes well- incorporated into” the district s activity cycle. 
As the required evaluations get underway, LEAs: feel that they are able | 
to expand their evaluation program to include other areas. bes - 
As an example of expanded evaluation activity, anurans on by the 
basic TIERS saaiviaene ts the State of lowa (with support from their TAC) ° 
2 | is developing a‘ broad evaluation program, including periodic special,’ 
| intensive studies of dhe senviees provided. The first step in this © 
process was an effort £5 ensure that the data generated were of . 
good qua lity. Once this was done, the next phase’was to identify | 
specific program variables one, seem to be associated with student. 
“achievement. After successful practices are identified,. they will he ‘ 
disseminated to LEAs so that they can be. ‘incorporated into plans to - 
improve the Title I reading program. In addition, the ‘study will provide 
. ‘the SEA with a detailed description ‘of the instruct tonal “Practices, methods 
_and meerals currently employed in the State. | 3 we ‘e 
In addition to the Evaluation Use Committee, and pelated work of 
_ the TACs, a second. major area of ecera support fp local and State 
use Of evaluation data is through x set of contracts for "State Refine- 
"ments to the USOE Title I Evaluation and Reporting system." In fiscal 


_ year 1979, fourteen contracts were awarded to support these ‘activities. 


- 
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“These awards fall into four main categories: improvement of eat 


collection and, analysis- activities, ‘development and dissemination of, 


new. materials, projects related to testing and evaluation nethodo} ogy, . | ice 


snd a miscellaneous category: “Three SEAs were awarded contracts to work 


in the "quality control" area == . cataloging errors. made in data collection; 


and evaluation methodology, including. efforts to examine ‘alternative 


- empirically some assumptions of the Title I nodel a “hes antes to ' ‘. 


| scoré -conversion, and analysis, and then implementing an evaluation program. - 


less susceptible to these: threats to validity. Five SEAS’ Fecet ved awards 


- to develop and disseminate new naterials, including ‘slide/tape presenta- 


tions, reference guides, software velopment and training and ‘implementa- ° 


tion matertals. eae SEAS are conduct ing projects related to nN 


methods of test equating dn Title I evaluation, ti devetop adequate 


measures of affective impact of Title, I services, 


local situations. In the next round of State Ref ment contracts ;USO§ 


is currently considering the faddition of © ‘several ew explicit categories “34 # 


for which awards could be made. One new catego, y may Fociig on- improving 


‘the utflity’ ‘of evaluation information at the 1 cal and SEA level's, and 


, another may examine the management éf ficiency 


| examine how Title I data are usm if at all, to improve Program 


‘isa the adoption 


of the USOE TIERS. For example, in the former category, am SEA Qe 


» design new program component's ‘to ensure that gains made in the regular 


. school year are sustajned (see Poly 95-561, sect fons. 124(9)(3) and atk). 


A third area of FéderaT ‘support for. jforoving the utility of Title. 1° 


‘evaluation informat fon at the SEA and LEA “levels is: through “the. (on-going 


development é minded aapiaorilta for eva luaping other, areas of 


Title, I, PAPE AGUIAR pre- second ae This. project 18 poy 
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in its fourth and final phase, which is fie actus) development of i? 
materials and the dissemination of strategies suitable for use at the 

pe early grades. In addition to the provision of analysis strategies which 

hay “prove usefdtsto examine the ig@pact of these programs on student 


achievement, substantial effort is being devoted to developing a system 


where utility at the local level is the prime factor. A system targeted 
at improving local programs, in which, for example, various methods for 
defining program objectives, sett ing up investigative and program review 
Cima FEEGS, conducting ‘on-site visitations, and collecting and ‘reporting 
on Panel nane. information, is being developed. A descriptive reporting. 

. system, lla aimed at providing information on the nature and. extent 
of offered ‘services. at the LEA and SEA levels 1s also being devel oped,. 
‘as isa system for dbsigning and implement ing longitudinal evaluations. 


Methods for selecting children to participate in early childhood 


Title I programs, “long a restive issue, i methods for evaluating parental 
dnvolvement component's of these programs, are also being investigated. ‘ 
It. ts expected that, ‘particularly in these two areas, these materials ' 
will have’ ultimate, ‘payoff across the spectrum ‘of the Title I program. 
, In summary, USOE 4s taking a more active rate in working with . 
LEAS, ms and “Federal contractors to help develop, refine and disseminate 
i ity-based luat ion procedures. A necessary first step was 
ta mplement t the TIERS, both in arder to meet a congressional mandate 
_and to prreids a sound basis for more extensive evaluation work. 
Now that the first goal is wel] on its way to being met, a larger 
proportion of activity will be Parganas toward: ae evaluation ‘infor-* 


mation for local and State program improvement. , 
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